02:14
2026-06-16
discuss.huggingface.co
developer-tools
Quicktok: A faster tokenizer
Developer dmatth1 released quicktok, a fast exact BPE tokenizer written in C++ that runs 2–3.5× faster than bpe-openai and 4–11× faster than tiktoken on CPU. The tokenizer is byte-identical to tiktoke…